auto_metrics Nightly Notebook

By Josh Dillon

Last Updated January 27, 2021

auto_metrics is a module in hera_qm that computes a series of statistics on night-long autocorrelation waterfalls to find antenna outliers in shape, power, or temporal structure. In general, these are assessed by collapsing each waterfall to a single (normalized) spectrum, then comparing that spectrum to the mean/median of the unflagged antennas' spectra to compute some difference metric. Those are then converted into modified Z-scores by comparing the overall distribution of good antennas and the worst antenna is flagged if it exceeds some threshold. This whole processes is repeated iteratively until no new bad antennas are identified. This proceeds in two rounds, first with more robust median-based statistics to identify the worst outliers, and then (after an RFI flagging step), a round with mean-based statistics. This notebook examines those mean-based spectra and statistics.

Statistics computed (after removing the worst offenders with a median-based metrics and then RFI flagging):

  • Shape: Compute the mean spectrum over time for each autocorrelation, but then divide by the mean of the whole waterfall to get something near 1. Now compute the mean absolute value of the difference between that and the mean spectrum over all non-excluded antennas of the same polarization. Convert this to a modified Z-score by comparing to all non-excluded antennas.
  • Power: Compute the mean spectrum over time for each autocorrelation without normalizing. Now compute the mean absolute value of the difference between the log of that (because we care about power outliers in dB rather than linear units) and the log of the mean spectrum over all non-excluded antennas of the same polarization. Convert this to a modified Z-score by comparing to all non-excluded antennas.
  • Temporal variability: Divide each autocorrelation by the mean waterfall for all non-excluded antennas of the same polarization. Now reduce to a single spectrum by computing the standard deviation along the time axis for each. Now compute the mean value (not the absolute value, since low variability shouldn't get a high Z-score) of the difference between that and the mean spectrum over all non-excluded antennas of the same polarization. Convert this to a modified Z-score by comparing to all non-excluded antennas.
  • Temporal discontinuities: Divide each autocorrelation by the mean waterfall for all non-excluded antennas of the same polarization. Now compute the element-by-element difference along the time axis, take the absolute value, and take the mean along the time axis to get a single spectrum. Now compute the mean value (not the absolute value, since low variability shouldn't get a high z-score) of the difference between that and the mean spectrum over all non-excluded antennas of the same polarization. Convert this to a modified Z-score by comparing to all non-excluded antennas.
In [1]:
import numpy as np
import pandas as pd
pd.set_option('display.max_rows', 1000)
import matplotlib.pyplot as plt
import matplotlib
import glob
import os
import operator
from hera_cal.io import HERAData
from hera_cal.utils import split_pol
from hera_qm.metrics_io import load_metric_file
from hera_notebook_templates.utils import status_colors
from IPython.display import display, HTML

%matplotlib inline
%config InlineBackend.figure_format = 'retina'

Parse Inputs and Load Data

In [2]:
# If you want to run this notebook locally, copy the output of the next cell into the first few lines of this cell.
# JD = '2459122'
# data_path = '/lustre/aoc/projects/hera/H4C/2459122'
# os.environ["JULIANDATE"] = JD
# os.environ["DATA_PATH"] = data_path
In [3]:
# Use environment variables to figure out path to data
JD = os.environ['JULIANDATE']
data_path = os.environ['DATA_PATH']
print(f'JD = "{JD}"')
print(f'data_path = "{data_path}"')
JD = "2459241"
data_path = "/users/jsdillon/lustre/H4C_RTP/auto_metrics_notebooks/2459241"
In [4]:
print(f'Looking for data in {data_path} on JD {JD}...')

auto_metrics_file = sorted(glob.glob(os.path.join(data_path, f'zen.{JD}*.auto_metrics.h5')))
if len(auto_metrics_file) > 0:
    auto_metrics_file = auto_metrics_file[0]
    print(f'Found auto_metrics results file at {auto_metrics_file}.')
else:
    raise OSError(f'{auto_metrics_file} not found.')

raw_auto_files = sorted(glob.glob(os.path.join(data_path, f'zen.{JD}.?????.sum.autos.uvh5')))
if len(raw_auto_files) > 0:
    print(f'Found {len(raw_auto_files)} extracted autocorrelation files.')
else: 
    raise OSError(f'No files of the form zen.{JD}.?????.sum.autos.uvh5 found in {data_path}.')
Looking for data in /users/jsdillon/lustre/H4C_RTP/auto_metrics_notebooks/2459241 on JD 2459241...
Found auto_metrics results file at /users/jsdillon/lustre/H4C_RTP/auto_metrics_notebooks/2459241/zen.2459241.49564.sum.auto_metrics.h5.
Found 600 extracted autocorrelation files.
In [5]:
# load auto_metrics and define use someful quantities
am = load_metric_file(auto_metrics_file)
mean_round_modz_cut = am['parameters']['mean_round_modz_cut']
ex_ants = am['ex_ants']['r2_ex_ants']
ants = sorted(set(bl[0] for bl in am['modzs']['r2_shape_modzs']))

# load raw autocorrelation waterfalls and define some useful quantities
hd = HERAData(raw_auto_files)
autos, _, _ = hd.read(axis='blt')
wf_shape = next(iter(autos.values())).shape
freqs = autos.freqs / 1e6
times = autos.times
lsts = autos.lsts * 12 / np.pi
In [6]:
# try to load a priori antenna statusesm but fail gracefully if this doesn't work.
a_priori_statuses = {ant: 'Not Found' for ant in ants}
try:
    from hera_mc import cm_active
    h = cm_active.ActiveData(at_date=JD)
    h.load_apriori()    
    for ant_name in h.apriori:
        ant = int("".join(filter(str.isdigit, ant_name)))
        if ant in a_priori_statuses:
            a_priori_statuses[ant] = h.apriori[ant_name].status
except Exception as err:
    print(f'Could not load a priori antenna statuses.\nEncountered {type(err)} with message: {err}')

Summary Plots and Tables

In [7]:
# print ex_ants for easy copy-pasting to YAML file
print('ex_ants: [' + ", ".join(str(ant) for ant in ex_ants) + ']')
ex_ants: [0, 1, 2, 11, 12, 13, 14, 23, 24, 25, 26, 39, 73, 82, 83, 84, 85, 86, 87, 92, 93, 94, 99, 102, 103, 112, 116, 121, 122, 123, 127, 128, 130, 136, 138, 143, 145, 155, 156, 157, 158, 165, 166, 178, 179, 184, 186, 187]
In [8]:
def Array_Plot():
    plt.figure(figsize=(8, 8), dpi=75)
    plt.scatter(np.array(list(autos.antpos.values()))[:,0], 
                np.array(list(autos.antpos.values()))[:,1], c='w', s=0)
    for ant, pos in autos.antpos.items():
        bad = ant in ex_ants
        plt.gca().add_artist(plt.Circle(tuple(pos[0:2]), radius=7, 
                                        fill=(~bad), color=['grey','r'][bad]))
        plt.text(pos[0],pos[1],str(ant), va='center', ha='center', color='w')
    plt.xlabel("Antenna East-West Position (meters)")
    plt.ylabel("Antenna North-South Position (meters)")
    plt.title(f'Antenna Positions and Auto_Metrics Flags on {JD}\n(Maximum Modified Z-Score > {mean_round_modz_cut} in Red)');
    plt.axis('equal')
    plt.tight_layout()

Figure 1: Antenna Positions with auto_metrics flags.

This plot shows the antenna positions of all antennas in the data. The antennas with at least one Modified Z-score for one metric on one polarization exceeding the cut are entirely flagged.

In [9]:
Array_Plot()
In [10]:
# Parse modzs for Table 1 and other figures
modzs_to_check = {'Shape': 'r2_shape_modzs', 'Power': 'r2_power_modzs', 
                  'Temporal Variability': 'r2_temp_var_modzs', 'Temporal Discontinuties': 'r2_temp_diff_modzs'}
worst_metrics = []
worst_zs = []
all_modzs = {}
binary_flags = {rationale: [] for rationale in modzs_to_check}

for ant in ants:
    # parse modzs and figure out flag counts
    modzs = {f'{pol} {rationale}': am['modzs'][dict_name][(ant, ant, pol)] 
             for rationale, dict_name in modzs_to_check.items() for pol in autos.pols()}
    for pol in autos.pols():
        for rationale, dict_name in modzs_to_check.items():
            binary_flags[rationale].append(am['modzs'][dict_name][(ant, ant, pol)] > mean_round_modz_cut)
    
    # figure out which metric is the largest outlier
    worst_metric, worst_z = max(modzs.items(), key=operator.itemgetter(1))
    worst_metrics.append(worst_metric)
    worst_zs.append(worst_z)
    
    # parse out all metrics for dataframe
    for k in modzs:
        col_label = k + ' Modified Z-Score'        
        if col_label in all_modzs:
            all_modzs[col_label].append(modzs[k])
        else:
            all_modzs[col_label] = [modzs[k]]
In [11]:
# build dataframe
to_show = {'Ant': ants, 'A Priori Status': [a_priori_statuses[ant] for ant in ants],
           'Worst Metric': worst_metrics, 'Worst Modified Z-Score': worst_zs}
to_show.update(all_modzs)
df = pd.DataFrame(to_show).sort_values('Worst Modified Z-Score', ascending=False)

# style dataframe
z_score_cols = [col for col in df.columns if col not in ['Ant', 'A Priori Status', 'Worst Metric']]
table1 = df.style.hide_index()\
           .applymap(lambda val: 'font-weight: bold' if val in ex_ants else '', subset=['Ant']) \
           .applymap(lambda val: 'color: red' if val in ex_ants else '', subset=['Ant']) \
           .applymap(lambda val: f'background-color: {status_colors[val]}' if val in status_colors else '', subset=['A Priori Status']) \
           .background_gradient(cmap='viridis', vmax=mean_round_modz_cut * 3, vmin=0, axis=None, subset=z_score_cols) \
           .applymap(lambda val: 'font-weight: bold' if val > am['parameters']['mean_round_modz_cut'] else '', subset=z_score_cols) \
           .applymap(lambda val: 'color: red' if val > am['parameters']['mean_round_modz_cut'] else '', subset=z_score_cols) \
           .set_table_styles([dict(selector="th",props=[('max-width', '70pt')])])

Table 1: Modified Z-Score Summary

This table displays the metrics for each antenna, highlighting which one is the worst. It is sorted by each antenna's worst metric. When one metric exceeds the threshold, auto_metrics recommends cutting that antenna. Flagged antennas and metrics exceeding the cut are shown in bold and red. Also shown is the antenna's a priori status.

In [12]:
HTML(table1.render())
Out[12]:
Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
2 digital_ok nn Temporal Discontinuties 957.360338 218.708179 200.969644 nan nan 340.098492 355.175441 893.925785 957.360338
1 digital_ok nn Temporal Discontinuties 594.633436 245.195833 243.380938 nan nan 197.450713 209.785975 462.110653 594.633436
24 RF_maintenance ee Temporal Discontinuties 574.474169 245.018976 244.271741 nan nan 296.486207 116.868460 574.474169 191.801791
0 digital_ok nn Temporal Discontinuties 563.666422 152.585461 178.062620 nan nan 232.118368 252.414925 523.758556 563.666422
12 digital_ok ee Temporal Discontinuties 559.091913 196.608200 211.053023 nan nan 215.464309 181.451623 559.091913 416.766827
23 digital_ok nn Temporal Discontinuties 460.206775 171.781560 130.936495 nan nan 191.140014 182.692016 445.652008 460.206775
14 digital_ok ee Temporal Discontinuties 457.840253 145.389900 219.453746 nan nan 161.741696 135.008051 457.840253 283.102536
13 digital_ok ee Temporal Discontinuties 447.371093 245.136755 243.858577 nan nan 199.957213 166.516925 447.371093 372.128423
11 RF_maintenance nn Temporal Discontinuties 402.019332 225.583839 229.455744 nan nan 113.810430 157.381189 241.884588 402.019332
39 digital_ok ee Temporal Discontinuties 401.332524 245.828697 244.653797 nan nan 219.027984 60.996751 401.332524 127.690776
127 RF_maintenance nn Temporal Discontinuties 273.403242 0.176953 18.163121 nan nan 42.860144 230.806469 54.315530 273.403242
25 digital_ok ee Temporal Discontinuties 253.274124 239.539763 238.011575 nan nan 122.173827 128.506237 253.274124 243.549488
121 RF_maintenance ee Shape 244.464650 244.464650 79.124300 nan nan 12.912001 10.421957 11.462025 9.587866
116 RF_maintenance ee Shape 244.419942 244.419942 54.467942 nan nan 14.284401 9.818087 14.266800 13.451352
73 digital_ok ee Shape 243.513258 243.513258 2.198064 nan nan 9.231876 3.439216 9.939316 3.282680
26 digital_ok ee Shape 243.202675 243.202675 241.207780 nan nan 66.675704 124.346775 98.425375 222.424273
143 RF_maintenance ee Shape 243.019521 243.019521 33.649090 nan nan 13.283471 14.317622 8.728143 2.527302
187 RF_ok ee Shape 242.888005 242.888005 4.560373 nan nan 15.976381 6.626446 9.642726 7.275936
103 digital_ok ee Shape 242.834635 242.834635 1.456357 nan nan 11.362716 4.199620 4.428074 2.365318
184 RF_maintenance ee Shape 242.392312 242.392312 1.642812 nan nan 8.851417 6.163667 4.538732 3.997486
158 digital_ok ee Shape 241.101007 241.101007 4.772039 nan nan 14.763461 9.311855 8.173550 5.691830
94 digital_ok ee Shape 240.018847 240.018847 3.446893 nan nan 15.707062 6.647983 17.389098 6.412490
122 digital_ok ee Shape 239.946326 239.946326 3.688063 nan nan 10.198859 4.939090 7.457128 3.260365
128 RF_maintenance ee Shape 238.974157 238.974157 1.982274 nan nan 20.608593 83.964322 16.913441 59.051713
112 digital_ok ee Temporal Discontinuties 236.371551 17.972673 8.974274 nan nan 209.955923 95.094947 236.371551 57.366545
155 digital_ok ee Shape 233.986771 233.986771 62.209188 nan nan 30.717115 13.894581 28.622253 19.596126
138 RF_maintenance ee Shape 229.074148 229.074148 0.940614 nan nan 10.253404 5.584823 5.237978 1.388560
178 RF_maintenance ee Shape 227.978351 227.978351 2.098333 nan nan 17.106392 5.845904 7.700945 3.271587
85 digital_ok nn Temporal Discontinuties 87.392862 3.172009 5.246044 nan nan 38.819166 45.848542 78.871812 87.392862
83 digital_ok nn Temporal Discontinuties 74.628660 38.581174 25.095954 nan nan 47.342507 65.681844 41.918252 74.628660
86 RF_maintenance ee Shape 70.286777 70.286777 1.181070 nan nan 43.876209 6.343176 28.743319 3.453012
82 digital_ok ee Temporal Discontinuties 66.344345 16.601417 4.163926 nan nan 56.656808 6.250149 66.344345 1.865128
136 RF_maintenance ee Shape 44.496504 44.496504 2.371125 nan nan 6.333910 8.388953 5.491893 4.228600
145 RF_ok ee Shape 40.356722 40.356722 1.414209 nan nan 5.435531 3.446225 6.390930 2.610162
123 RF_maintenance ee Shape 40.210527 40.210527 2.745327 nan nan 4.776806 14.827072 2.181748 1.146953
99 digital_ok ee Shape 40.183655 40.183655 0.496258 nan nan 4.343476 4.832494 3.030294 4.549701
130 RF_maintenance ee Shape 39.977359 39.977359 2.781969 nan nan 5.235431 4.612593 6.095514 3.229517
179 RF_maintenance ee Shape 38.023413 38.023413 3.020600 nan nan 6.164557 11.385911 8.256993 3.131860
92 digital_ok ee Shape 30.009990 30.009990 1.141539 nan nan 4.096717 5.195577 5.986499 3.120968
84 RF_maintenance ee Temporal Variability 13.239870 4.341490 2.137635 nan nan 13.239870 4.666114 -0.477540 0.032732
93 digital_ok nn Temporal Variability 13.089688 1.209921 0.900232 nan nan 5.695525 13.089688 1.764953 2.649696
156 digital_ok ee Temporal Variability 8.762223 1.016939 0.998874 nan nan 8.762223 8.426225 1.697652 1.336009
166 RF_maintenance nn Shape 8.580063 0.796045 8.580063 nan nan -0.098498 5.441070 2.040182 1.222290
157 digital_ok nn Temporal Variability 7.046823 0.409743 0.202029 nan nan 4.701064 7.046823 1.998779 2.957086
102 digital_ok ee Temporal Variability 6.446640 0.806253 2.487353 nan nan 6.446640 2.652238 5.518103 0.206981
165 RF_ok nn Temporal Variability 6.424569 2.541349 1.586134 nan nan 0.920524 6.424569 0.107955 1.016435
186 RF_ok ee Temporal Variability 4.526057 0.514010 -0.107016 nan nan 4.526057 2.933643 4.081045 2.782053
87 RF_maintenance nn Temporal Variability 3.706813 0.611440 1.587209 nan nan 2.057291 3.706813 0.627124 2.388844
164 RF_ok ee Shape 3.639473 3.639473 -0.845487 nan nan 2.978253 0.437875 -0.500606 -0.795904
81 digital_ok ee Shape 3.382530 3.382530 -0.491322 nan nan 1.525392 -1.186511 -1.516180 -0.667583
98 digital_ok nn Shape 3.301441 0.730136 3.301441 nan nan 0.485975 1.931129 2.276622 0.741681
104 digital_ok ee Shape 2.522715 2.522715 -0.229012 nan nan 0.111635 -0.970557 -0.655145 -0.924050
129 dish_maintenance ee Temporal Variability 1.980413 0.761800 1.368276 nan nan 1.980413 0.683153 1.473301 1.470355
45 digital_ok ee Temporal Discontinuties 1.903096 0.618845 -0.139606 nan nan -0.514893 0.773368 1.903096 0.894425
46 digital_ok nn Temporal Discontinuties 1.282735 0.078866 0.463173 nan nan -0.578157 0.536861 0.478224 1.282735
100 digital_ok nn Temporal Variability 1.212363 0.584938 0.038378 nan nan 0.699945 1.212363 -0.800894 -0.130932
176 RF_maintenance nn Temporal Discontinuties 1.134960 -0.022819 0.022819 nan nan -0.743770 -0.124540 0.607557 1.134960
135 digital_ok nn Temporal Variability 1.133899 -0.297110 -1.018636 nan nan 0.016304 1.133899 0.048846 0.583098
119 digital_ok nn Shape 1.099433 0.560515 1.099433 nan nan -0.300831 0.665829 -1.402978 -1.080845
101 digital_ok ee Shape 0.975014 0.975014 -1.039244 nan nan -1.614246 -0.518556 0.354550 -0.229976
185 RF_ok ee Temporal Discontinuties 0.906438 -0.585395 -0.090690 nan nan -0.016304 0.097339 0.906438 0.681398
177 digital_ok ee Temporal Variability 0.595537 -0.888217 -1.105990 nan nan 0.595537 -0.486929 -0.164407 0.183098
144 RF_ok nn Shape 0.252213 -0.865768 0.252213 nan nan -1.214420 -0.222302 -0.310348 0.086809
163 RF_ok ee Temporal Discontinuties -0.048846 -0.306297 -1.110194 nan nan -0.465126 -0.958904 -0.048846 -0.145945
120 RF_maintenance nn Shape -0.201228 -1.388972 -0.201228 nan nan -1.178389 -1.236198 -0.594058 -1.028151
In [13]:
def Flag_Bar_Chart():
    plt.figure(figsize=(8, 4), dpi=100)

    # count 
    rationales = list(binary_flags.keys())
    flags_list = np.array([binary_flags[rationale] for rationale in rationales])
    antpol_flags = [np.sum(f) for f in flags_list]
    ant_flags = [np.sum(np.array(f)[0::2] | np.array(f)[1::2]) for f in flags_list]

    # make bar chart
    plt.bar(np.arange(len(rationales)), antpol_flags, width=.7, color='yellow', ec='k', tick_label=rationales)
    for x, (nflags, nants) in enumerate(zip(antpol_flags, ant_flags)):
        plt.text(x, nflags/2, f'{nflags} Feeds on \n{nants} Antennas\nFlagged', va='center', ha='center')

    # set labels
    plt.ylabel('Antenna-Polarizations Flagged')
    plt.xlabel('Reason for Flagging')
    plt.tight_layout()

Figure 2: Flagging Rationale Summary

This bar chart summarizes the number of antenna-polarizations that are statistical outliers in each metric (though often they overlap). Some of these issues occur on both polarizations, so there are fewer unique antennas flagged for each rationale than there are ant-pols flagged, as noted by the labels.

In [14]:
Flag_Bar_Chart()
In [15]:
def Rationale_Corr_Plot():
    plt.figure(figsize=(6,6), dpi=100)
    
    # compute correlation matrix
    rationales = list(binary_flags.keys())
    flags_list = np.array([binary_flags[rationale] for rationale in rationales])
    corrs = np.corrcoef(flags_list)
    
    # plot and label correlation matrix
    plt.imshow(corrs, cmap='viridis', interpolation='nearest', origin='upper')
    for i in range(corrs.shape[0]):
        for j in range(corrs.shape[1]):
            plt.text(i, j, np.round(corrs[i, j], 3), va='center', ha='center',
                     bbox={'facecolor': 'w', 'ec': 'w', 'alpha': .75})

    # colorbar, labels, and style
    plt.yticks(range(len(rationales)), rationales)
    plt.xticks(range(len(rationales)), rationales, rotation=-45, ha='right')
    plt.gca().xaxis.tick_top()
    plt.clim([0, 1])
    plt.colorbar(fraction=0.046, pad=0.04)
    plt.tight_layout()

Figure 3: Flagging Rationale Correlations

This plot shows the probability that if a given ant-pol is flagged for some reason, it's also flagged for another reason.

In [16]:
Rationale_Corr_Plot()
In [17]:
def plot_all_spectra(spectra, mod_zs, modz_cut, overall, freqs, reason, ex_ants=[], 
                     xlabel='Frequency (MHz)', ylabel='', yscale='linear', ylim_factor=None):
    '''Helper function for plotting all spectra and showing which ones were flagged and why.''' 
    fig, axes = plt.subplots(1,2, figsize=(14,5), dpi=100)
    pols = sorted(set([bl[2] for bl in spectra]))
    for ax, pol in zip(axes, pols):
        # sort antennas into good, bad, and bad but not for this reason
        bad_here = [bl for bl in spectra if (bl[2] == pol) and mod_zs[bl] > modz_cut]
        other_bad = [bl for bl in spectra if (bl[2] == pol) and (bl[0] in ex_ants) and (bl not in bad_here)]
        good = [bl for bl in spectra if (bl[2] == pol) and (bl[0] not in ex_ants)]
        
        # plot all spectra
        l1, l2, l3 = None, None, None
        for bl in other_bad:
            l2, = ax.plot(freqs, spectra[bl], 'darkviolet', lw=.5)
        for bl in bad_here:
            l1, = ax.plot(freqs, spectra[bl], 'r', lw=.5)
        for bl in good:
            l3, = ax.plot(freqs, spectra[bl], 'grey', alpha=.5)
        l4, = ax.plot(freqs, overall[bl[2]], 'k--')
        
        ax.set_xlabel(xlabel)
        ax.set_ylabel(ylabel)
        ax.set_yscale(yscale)
        if ylim_factor is not None:
            ax.set_ylim([np.nanmin([spectra[bl] for bl in good]) / ylim_factor, 
                         np.nanmax([spectra[bl] for bl in good]) * ylim_factor])
        ax.set_title(f'Outliers in Autocorrelation {reason}: {pol}')
        ax.legend([l1, l2, l3, l4], [f'Flagged for {reason}', 'Flagged for Another Reason', 'Unflagged', 'Average Unflagged'], loc='lower right')
    plt.tight_layout()
In [18]:
overall_shape = {pol: np.nanmean([spec for bl, spec in am['spectra']['mean_spectra_normed'].items() if (bl[2] == pol) 
                                  and (bl[0] not in ex_ants)], axis=0) for pol in autos.pols()}
overall_power = {pol: np.nanmean([spec for bl, spec in am['spectra']['mean_spectra'].items() if (bl[2] == pol) 
                                  and (bl[0] not in ex_ants)], axis=0) for pol in autos.pols()}
overall_temp_var = {pol: np.nanmean([spec for bl, spec in am['spectra']['std_spectra_normed'].items() if (bl[2] == pol) 
                                     and (bl[0] not in ex_ants)], axis=0) for pol in autos.pols()}
overall_temp_diff = {pol: np.nanmean([spec for bl, spec in am['spectra']['mean_abs_diff_spectra_normed'].items() if (bl[2] == pol) 
                                      and (bl[0] not in ex_ants)], axis=0) for pol in autos.pols()}

Figure 4: Outliers in Autocorrelation Shape

This plot summarizes the spectra computed to compare to one another to find outliers in autocorrelation shape (see above for how that was computed). The mean compared to is shown as a black dashed line. Antennas in red were flagged as outliers, antennas in gray and purple were not. However, antennas in purple were flagged for some other reason, either another metric or on the other polarization. Completely flagged channels (RFI and band edges) appear as white gaps.

In [19]:
plot_all_spectra(am['spectra']['mean_spectra_normed'], am['modzs']['r2_shape_modzs'], mean_round_modz_cut, 
                 overall_shape, freqs, 'Shape', ex_ants=ex_ants, yscale='linear', ylim_factor=1.2)

Figure 5: Outliers in Autocorrelation Power

This plot summarizes the spectra computed to compare to one another to find outliers in autocorrelation amplitude (see above for how that was computed). The mean compared to is shown as a black dashed line. Antennas in red were flagged as outliers, antennas in gray and purple were not. However, antennas in purple were flagged for some other reason, either another metric or on the other polarization. Completely flagged channels (RFI and band edges) appear as white gaps.

In [20]:
plot_all_spectra(am['spectra']['mean_spectra'], am['modzs']['r2_power_modzs'], mean_round_modz_cut, 
                 overall_power, freqs, 'Power', ex_ants=ex_ants, yscale='log')

Figure 6: Outliers in Autocorrelation Temporal Variability

This plot summarizes the spectra computed to compare to one another to find outliers in autocorrelation temporal variability (as measured by a standard deviation over time; see above for how that was computed). The mean compared to is shown as a black dashed line. Antennas in red were flagged as outliers, antennas in gray and purple were not. However, antennas in purple were flagged for some other reason, either another metric or on the other polarization. Completely flagged channels (RFI and band edges) appear as white gaps.

In [21]:
plot_all_spectra(am['spectra']['std_spectra_normed'], am['modzs']['r2_temp_var_modzs'], mean_round_modz_cut,
                 overall_temp_var, freqs, 'Temporal Variability', ex_ants=ex_ants, yscale='log')

Figure 7: Outliers in Autocorrelation Temporal Discontinuities

This plot summarizes the spectra computed to compare to one another to find outliers in autocorrelation temporal discontinuities (as measured by the average absolute integration-to-integration difference over time; see above for how that was computed). The mean compared to is shown as a black dashed line. Antennas in red were flagged as outliers, antennas in gray and purple were not. However, antennas in purple were flagged for some other reason, either another metric or on the other polarization. Completely flagged channels (RFI and band edges) appear as white gaps.

In [22]:
plot_all_spectra(am['spectra']['mean_abs_diff_spectra_normed'], am['modzs']['r2_temp_diff_modzs'], mean_round_modz_cut, 
                 overall_temp_diff, freqs, 'Temporal Discontinutities', ex_ants=ex_ants, yscale='log')
In [23]:
# compute average good autocorrelations for each polarization
avg_good_autos = {pol: np.zeros(wf_shape, dtype=float) for pol in autos.pols()}
for pol in autos.pols():
    for i in range(wf_shape[0]):
        avg_good_autos[pol][i] = np.mean([np.abs(autos[bl][i, :]) for bl in autos 
                                          if (bl[0] not in ex_ants) and (bl[2] == pol)], axis=0)
    avg_good_autos[pol][am['flags']] = np.nan
In [24]:
def Avg_Auto_Plot():
    fig, axes = plt.subplots(1, 2, figsize=(14,5), dpi=100)
    for ax, pol in zip(axes, autos.pols()):
        im = ax.imshow(avg_good_autos[pol], aspect='auto', interpolation='nearest',
                       extent=[freqs[0], freqs[-1], times[-1], times[0]])
        ax.set_yticklabels(np.around(lsts[[min(max(np.searchsorted(times, t), 0), len(times) - 1) for t in ax.get_yticks()]], 2))    
        plt.colorbar(im, ax=ax)
        ax.set_title(f'Average Good Raw {pol} Autocorrelation After Flagging')
        ax.set_ylabel('LST (hours)')
        ax.set_xlabel('Frequency (MHz)')
    plt.tight_layout()

Figure 8: Average Good Autocorrelations and Flags

Here we show the waterfalls of the array-averaged autocorrelations over the night, after removing all flagged antennas. We also show the RFI mask generated between the median and mean rounds of antenna outlier detection. This is meant to show that there is little or no RFI remaining to affect the statistics.

In [25]:
Avg_Auto_Plot()

Per-Antenna Plots

In [26]:
def plot_spectra(axes, ant, spec, modzs, modz_cut, overall, reason, yscale='linear'):
    '''Helper function for plotting both antennas of a given polarization.'''
    for pol, ax in zip(autos.pols(), axes):
        bl = (ant, ant, pol)

        # plot good antennas
        for bl2 in modzs:
            if (bl2[0] not in ex_ants) and (bl2[2] == pol):
                ax.plot(freqs, spec[bl2], 'grey', lw=.5, alpha=.5)
        ax.plot(freqs, overall[pol], 'k--')

        # plot this anetnna
        color = 'r'
        alpha = .75
        if modzs[bl] >= modz_cut:
            alpha = 1
        elif bl[0] in ex_ants:
            color = 'darkviolet'
        else:
            color = 'darkgreen'
        ax.plot(freqs, spec[bl], color, alpha=alpha, label=f'{ant} {pol} (z = {np.round(modzs[bl],1)})')

        # decorate axis
        ax.set_yscale(yscale)
        ax.legend(loc=4)
        ax.set_title(f'{pol} {reason}')  
In [27]:
len(ants)
Out[27]:
65
In [28]:
def Plot_All_Auto_Metrics():
    for row, ant in enumerate(df['Ant']): 
        # print title of section
        display(HTML(f'<h2>Antenna {ant}:</h2>'))

        # print metrics
        df_row = df.loc[df['Ant'] == ant]
        df_row = df_row.style.hide_index()\
                       .set_table_styles([dict(selector="th",props=[('max-width', '70pt')])])\
                       .applymap(lambda val: f'background-color: {status_colors[val]}' if val in status_colors else '', subset=['A Priori Status']) \
                       .applymap(lambda val: 'font-weight: bold' if val in ex_ants else '', subset=['Ant']) \
                       .applymap(lambda val: 'color: red' if val in ex_ants else '', subset=['Ant']) \
                       .applymap(lambda val: 'font-weight: bold' if val > am['parameters']['mean_round_modz_cut'] else '', subset=z_score_cols) \
                       .applymap(lambda val: 'color: red' if val > am['parameters']['mean_round_modz_cut'] else '', subset=z_score_cols)
        display(HTML(df_row.render()))

        # plot spectra and waterfalls
        fig, axes = plt.subplots(2, 6, figsize=(18, 5.5), dpi=(50 * (104 / len(ants))**.5)) # this should help manage filesize

        # plot individual spectra compared to all good antennas
        plot_spectra(axes[:, 0], ant, am['spectra']['mean_spectra_normed'], am['modzs']['r2_shape_modzs'], 
                     mean_round_modz_cut, overall_shape, 'Shape', yscale='linear')
        plot_spectra(axes[:, 1], ant, am['spectra']['mean_spectra'], am['modzs']['r2_power_modzs'], 
                     mean_round_modz_cut, overall_power, 'Power', yscale='log')
        plot_spectra(axes[:, 2], ant, am['spectra']['std_spectra_normed'], am['modzs']['r2_temp_var_modzs'],
                     mean_round_modz_cut, overall_temp_var, 'Temporal Variability', yscale='linear')
        plot_spectra(axes[:, 3], ant, am['spectra']['mean_abs_diff_spectra_normed'], am['modzs']['r2_temp_diff_modzs'],
                     mean_round_modz_cut, overall_temp_diff, 'Temporal Discontinutities', yscale='linear')
        plt.tight_layout()

        # plot linear-scale waterfalls 
        for pol, ax in zip(autos.pols(), axes[:, 4]):
            bl = (ant, ant, pol)        
            im = ax.imshow(np.where(am['flags'], np.nan, autos[bl].real), 
                           aspect='auto', interpolation='nearest',
                           extent=[freqs[0], freqs[-1], times[-1], times[0]])
            ax.set_yticklabels(np.around(lsts[[min(max(np.searchsorted(times, t), 0), len(times) - 1) for t in ax.get_yticks()]], 2))
            ax.set_title(f'{pol} Waterfall (Linear Scale)')

        # plot log-scale mean-divided waterfalls
        for pol, ax in zip(autos.pols(), axes[:, 5]):
            bl = (ant, ant, pol)
            to_plot = autos[bl].real / avg_good_autos[pol]
            to_plot[am['flags']] = np.nan
            to_plot /= np.nanmean(to_plot)
            im = ax.imshow(np.log10(to_plot), aspect='auto', interpolation='nearest', vmin=-.07, vmax=.07,
                           extent=[freqs[0], freqs[-1], times[-1], times[0]])
            ax.set_yticklabels(np.around(lsts[[min(max(np.searchsorted(times, t), 0), len(times) - 1) for t in ax.get_yticks()]], 2))
            ax.set_title(f'{pol} Log(Normalized Waterfall)')

        plt.show()

        # print some whitespace
        display(HTML('<hr style="height:3px">'))

Figure 9: Per-Antenna Statistics, Spectra, and Waterfalls

Here we show the metrics for each antenna and the spectra/waterfalls that hopefully explain what led to them. The table reproduces the information from Table 1 above. The first four panels in each row clearly highlight the antenna's spectrum as it compares to the mean good antenna (black) and the distribution of good antennas (gray). Spectra in red were flagged as outliers. Spectra in purple were flagged for some other reason, either another metric or on the other polarization. Good antennas are shown in green. Completely flagged channels (RFI and band edges) appear as white gaps. In the fifth column, the waterfall of that autocorrelation is shown on a linear scale after RFI/band edge flags (white). In the sixth column, we show the log (base 10) of the same waterfall, divided by the average good antennas' waterfall of that polarization and then normalized to an average of 1.

In [29]:
Plot_All_Auto_Metrics()

Antenna 2:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
2 digital_ok nn Temporal Discontinuties 957.360338 218.708179 200.969644 nan nan 340.098492 355.175441 893.925785 957.360338

Antenna 1:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
1 digital_ok nn Temporal Discontinuties 594.633436 245.195833 243.380938 nan nan 197.450713 209.785975 462.110653 594.633436

Antenna 24:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
24 RF_maintenance ee Temporal Discontinuties 574.474169 245.018976 244.271741 nan nan 296.486207 116.868460 574.474169 191.801791

Antenna 0:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
0 digital_ok nn Temporal Discontinuties 563.666422 152.585461 178.062620 nan nan 232.118368 252.414925 523.758556 563.666422

Antenna 12:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
12 digital_ok ee Temporal Discontinuties 559.091913 196.608200 211.053023 nan nan 215.464309 181.451623 559.091913 416.766827

Antenna 23:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
23 digital_ok nn Temporal Discontinuties 460.206775 171.781560 130.936495 nan nan 191.140014 182.692016 445.652008 460.206775

Antenna 14:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
14 digital_ok ee Temporal Discontinuties 457.840253 145.389900 219.453746 nan nan 161.741696 135.008051 457.840253 283.102536

Antenna 13:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
13 digital_ok ee Temporal Discontinuties 447.371093 245.136755 243.858577 nan nan 199.957213 166.516925 447.371093 372.128423

Antenna 11:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
11 RF_maintenance nn Temporal Discontinuties 402.019332 225.583839 229.455744 nan nan 113.810430 157.381189 241.884588 402.019332

Antenna 39:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
39 digital_ok ee Temporal Discontinuties 401.332524 245.828697 244.653797 nan nan 219.027984 60.996751 401.332524 127.690776

Antenna 127:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
127 RF_maintenance nn Temporal Discontinuties 273.403242 0.176953 18.163121 nan nan 42.860144 230.806469 54.315530 273.403242

Antenna 25:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
25 digital_ok ee Temporal Discontinuties 253.274124 239.539763 238.011575 nan nan 122.173827 128.506237 253.274124 243.549488

Antenna 121:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
121 RF_maintenance ee Shape 244.464650 244.464650 79.124300 nan nan 12.912001 10.421957 11.462025 9.587866

Antenna 116:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
116 RF_maintenance ee Shape 244.419942 244.419942 54.467942 nan nan 14.284401 9.818087 14.266800 13.451352

Antenna 73:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
73 digital_ok ee Shape 243.513258 243.513258 2.198064 nan nan 9.231876 3.439216 9.939316 3.282680

Antenna 26:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
26 digital_ok ee Shape 243.202675 243.202675 241.207780 nan nan 66.675704 124.346775 98.425375 222.424273

Antenna 143:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
143 RF_maintenance ee Shape 243.019521 243.019521 33.649090 nan nan 13.283471 14.317622 8.728143 2.527302

Antenna 187:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
187 RF_ok ee Shape 242.888005 242.888005 4.560373 nan nan 15.976381 6.626446 9.642726 7.275936

Antenna 103:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
103 digital_ok ee Shape 242.834635 242.834635 1.456357 nan nan 11.362716 4.199620 4.428074 2.365318

Antenna 184:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
184 RF_maintenance ee Shape 242.392312 242.392312 1.642812 nan nan 8.851417 6.163667 4.538732 3.997486

Antenna 158:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
158 digital_ok ee Shape 241.101007 241.101007 4.772039 nan nan 14.763461 9.311855 8.173550 5.691830

Antenna 94:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
94 digital_ok ee Shape 240.018847 240.018847 3.446893 nan nan 15.707062 6.647983 17.389098 6.412490

Antenna 122:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
122 digital_ok ee Shape 239.946326 239.946326 3.688063 nan nan 10.198859 4.939090 7.457128 3.260365

Antenna 128:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
128 RF_maintenance ee Shape 238.974157 238.974157 1.982274 nan nan 20.608593 83.964322 16.913441 59.051713

Antenna 112:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
112 digital_ok ee Temporal Discontinuties 236.371551 17.972673 8.974274 nan nan 209.955923 95.094947 236.371551 57.366545

Antenna 155:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
155 digital_ok ee Shape 233.986771 233.986771 62.209188 nan nan 30.717115 13.894581 28.622253 19.596126

Antenna 138:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
138 RF_maintenance ee Shape 229.074148 229.074148 0.940614 nan nan 10.253404 5.584823 5.237978 1.388560

Antenna 178:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
178 RF_maintenance ee Shape 227.978351 227.978351 2.098333 nan nan 17.106392 5.845904 7.700945 3.271587

Antenna 85:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
85 digital_ok nn Temporal Discontinuties 87.392862 3.172009 5.246044 nan nan 38.819166 45.848542 78.871812 87.392862

Antenna 83:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
83 digital_ok nn Temporal Discontinuties 74.628660 38.581174 25.095954 nan nan 47.342507 65.681844 41.918252 74.628660

Antenna 86:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
86 RF_maintenance ee Shape 70.286777 70.286777 1.181070 nan nan 43.876209 6.343176 28.743319 3.453012

Antenna 82:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
82 digital_ok ee Temporal Discontinuties 66.344345 16.601417 4.163926 nan nan 56.656808 6.250149 66.344345 1.865128

Antenna 136:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
136 RF_maintenance ee Shape 44.496504 44.496504 2.371125 nan nan 6.333910 8.388953 5.491893 4.228600

Antenna 145:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
145 RF_ok ee Shape 40.356722 40.356722 1.414209 nan nan 5.435531 3.446225 6.390930 2.610162

Antenna 123:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
123 RF_maintenance ee Shape 40.210527 40.210527 2.745327 nan nan 4.776806 14.827072 2.181748 1.146953

Antenna 99:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
99 digital_ok ee Shape 40.183655 40.183655 0.496258 nan nan 4.343476 4.832494 3.030294 4.549701

Antenna 130:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
130 RF_maintenance ee Shape 39.977359 39.977359 2.781969 nan nan 5.235431 4.612593 6.095514 3.229517

Antenna 179:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
179 RF_maintenance ee Shape 38.023413 38.023413 3.020600 nan nan 6.164557 11.385911 8.256993 3.131860

Antenna 92:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
92 digital_ok ee Shape 30.009990 30.009990 1.141539 nan nan 4.096717 5.195577 5.986499 3.120968

Antenna 84:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
84 RF_maintenance ee Temporal Variability 13.239870 4.341490 2.137635 nan nan 13.239870 4.666114 -0.477540 0.032732

Antenna 93:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
93 digital_ok nn Temporal Variability 13.089688 1.209921 0.900232 nan nan 5.695525 13.089688 1.764953 2.649696

Antenna 156:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
156 digital_ok ee Temporal Variability 8.762223 1.016939 0.998874 nan nan 8.762223 8.426225 1.697652 1.336009

Antenna 166:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
166 RF_maintenance nn Shape 8.580063 0.796045 8.580063 nan nan -0.098498 5.441070 2.040182 1.222290

Antenna 157:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
157 digital_ok nn Temporal Variability 7.046823 0.409743 0.202029 nan nan 4.701064 7.046823 1.998779 2.957086

Antenna 102:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
102 digital_ok ee Temporal Variability 6.446640 0.806253 2.487353 nan nan 6.446640 2.652238 5.518103 0.206981

Antenna 165:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
165 RF_ok nn Temporal Variability 6.424569 2.541349 1.586134 nan nan 0.920524 6.424569 0.107955 1.016435

Antenna 186:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
186 RF_ok ee Temporal Variability 4.526057 0.514010 -0.107016 nan nan 4.526057 2.933643 4.081045 2.782053

Antenna 87:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
87 RF_maintenance nn Temporal Variability 3.706813 0.611440 1.587209 nan nan 2.057291 3.706813 0.627124 2.388844

Antenna 164:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
164 RF_ok ee Shape 3.639473 3.639473 -0.845487 nan nan 2.978253 0.437875 -0.500606 -0.795904

Antenna 81:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
81 digital_ok ee Shape 3.382530 3.382530 -0.491322 nan nan 1.525392 -1.186511 -1.516180 -0.667583

Antenna 98:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
98 digital_ok nn Shape 3.301441 0.730136 3.301441 nan nan 0.485975 1.931129 2.276622 0.741681

Antenna 104:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
104 digital_ok ee Shape 2.522715 2.522715 -0.229012 nan nan 0.111635 -0.970557 -0.655145 -0.924050

Antenna 129:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
129 dish_maintenance ee Temporal Variability 1.980413 0.761800 1.368276 nan nan 1.980413 0.683153 1.473301 1.470355

Antenna 45:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
45 digital_ok ee Temporal Discontinuties 1.903096 0.618845 -0.139606 nan nan -0.514893 0.773368 1.903096 0.894425

Antenna 46:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
46 digital_ok nn Temporal Discontinuties 1.282735 0.078866 0.463173 nan nan -0.578157 0.536861 0.478224 1.282735

Antenna 100:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
100 digital_ok nn Temporal Variability 1.212363 0.584938 0.038378 nan nan 0.699945 1.212363 -0.800894 -0.130932

Antenna 176:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
176 RF_maintenance nn Temporal Discontinuties 1.134960 -0.022819 0.022819 nan nan -0.743770 -0.124540 0.607557 1.134960

Antenna 135:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
135 digital_ok nn Temporal Variability 1.133899 -0.297110 -1.018636 nan nan 0.016304 1.133899 0.048846 0.583098

Antenna 119:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
119 digital_ok nn Shape 1.099433 0.560515 1.099433 nan nan -0.300831 0.665829 -1.402978 -1.080845

Antenna 101:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
101 digital_ok ee Shape 0.975014 0.975014 -1.039244 nan nan -1.614246 -0.518556 0.354550 -0.229976

Antenna 185:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
185 RF_ok ee Temporal Discontinuties 0.906438 -0.585395 -0.090690 nan nan -0.016304 0.097339 0.906438 0.681398

Antenna 177:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
177 digital_ok ee Temporal Variability 0.595537 -0.888217 -1.105990 nan nan 0.595537 -0.486929 -0.164407 0.183098

Antenna 144:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
144 RF_ok nn Shape 0.252213 -0.865768 0.252213 nan nan -1.214420 -0.222302 -0.310348 0.086809

Antenna 163:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
163 RF_ok ee Temporal Discontinuties -0.048846 -0.306297 -1.110194 nan nan -0.465126 -0.958904 -0.048846 -0.145945

Antenna 120:

Ant A Priori Status Worst Metric Worst Modified Z-Score ee Shape Modified Z-Score nn Shape Modified Z-Score ee Power Modified Z-Score nn Power Modified Z-Score ee Temporal Variability Modified Z-Score nn Temporal Variability Modified Z-Score ee Temporal Discontinuties Modified Z-Score nn Temporal Discontinuties Modified Z-Score
120 RF_maintenance nn Shape -0.201228 -1.388972 -0.201228 nan nan -1.178389 -1.236198 -0.594058 -1.028151

In [30]:
from hera_qm import version
print(version.construct_version_info())
{'version': '1.0', 'git_origin': 'git@github.com:HERA-Team/hera_qm.git', 'git_hash': 'fe57ccbd3618c92d5a5db743e37d56a8cb629445', 'git_description': 'v1.1-378-gfe57ccb', 'git_branch': 'master'}
In [ ]: